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ABSTRACT 

This paper describes and evaluates the coverage and 
content validity of the vocational/career education tests available 
commercially to high school educators. Tests were analyzed with 
respect to their collective coverage of the vocational/career 
education curriculum goals appropriate to grades 11 and 12, and the 
extent to which each possessed good content validity as defined by 
two criteria* The results indicate a good overall coverage of the 
curriculum goals for secondary education, however the poor content 
validity of the majority of tests is disappointing. The results of 
this analysis have many implicaticTis for the improvement of 
vocational/career education tests. (Author) 
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Abstract 

This paper rfriscribes and evaluates the coverage and content validity 
of the vocational/career education tests available commercially to high school 
educators. Tests were analyzed with respect to their crllective coverage of 
the vocational/career education curriculum goals appropriate to grades 11 & 
12, and the extent to which each possessed good content validity as defined 
by two criteria* The results indicate a good overall coverage of the curriculum 
goals for secondary education, however the poor content validity of the majority 
of tests is disappointing. The results of this analysis have many implications 
for the improvement of vocational/career education tests. 
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COVERAGE AND CONTENT VALIDITY OF VOCATIONAL/CAREER EDUCATION TESTS 

Vocational and career education testing has traditionally been oriented 
toward career guidance in the schools. Much of the vocational testing in the 
high schools is conducted by counselors and psychologists who use results to 
help their students make realistic and satisfying career decisions regarding 
their futures. If tests are to have an impact on career exploration and 
development, counselors must have complete assurance that a test they have 
selected actually measures what it purports to measure. But because counselors 
are expected to be knowledgeable about many aspects of educational, vocational, 
and personal guidance, it is unrealistic to expect that they be experts in 
measurement as well. 

Publishers must assume a responsibility for assisting counselors to select 
the correct test for the appropriate purpose. It is no longer satisfactory for 
a test publisher to state that the test measures a goal of education; he must 
specify the goal, its limitations and extensions, and its manifestations in terms 
of behaviors to be learned. Then the publisher should present some documenta- 
tion that his claims are in fact supported. In fact, publishers should provide 
information about tests which is as complete and accurate as possible in order' 
to enable counselors to select tests best suited to their purposes. 

In response to this concern, the Center for the Study of Evaluation at 
U.C.L.A. undertook a comprehensive test evaluation project which included 
1) a survey of the vocational/career education tests that are commercially 
available for use with high school sutdents in grades 11 and 12, and 2) an 
evaluation of their adequacy with respect to content validity ( CSE Secondary 
School Test Evaluations: Grades 11 and 12 , Hoepfner et al., 1974). 
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Method 

In order to assess the availability and content appropriateness of vocational/ 
career education tests designed for use with 11th and 12th grade students, the 
Center for the Study of Evaluation first developed a comprehensive taxonomy 
of educational goals appropriate to secondary school education. Each goal cate- 
gory consists of a range of behaviors, achievemex.ts, or objectives. Only the 
portion of this taxonomy that pertained to vocational and career education was 
used for this paper. There are four major goal categories defined in the taxonomy 
under the heading of vocational/career education: General Knowledge and Skills 
for Vocational Success, General Vocational Aptitudes, General Vocational Atti- 
tudes, and General Vocational Interests. Subsumed under each of these four major 
goal categories is a total of 25 specific, behaviorally described, vocational/ 
career education goals. The names of the goal categories are listed in Table 1. 

The beginning step of the evaluation process was to assign each test and 
subtest to the particular goal category in the taxonomy into which it best fit. 
Judgments .were made on the basis of careful reading of the items in each test 
and assignment of each item to a specific goal category. 

After assignments to goal categories were completed, test evaluations were 
made according to predetermined standards. The evaluations of content validity 
were made according to two criteria. The first criterion addressed the question: 
what percentage of the range of behaviors, achievements, or objectives in the 
goal category to which the test was assigned do the items assess? To estimate 
this percentage, the total set of items of the test was compared to the range 
of behaviors in the goal description. Rater judgments credited the test with one 
point if 50% or more of the goal was deemed to be assessed and zero points 
otherwise. 
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The second criterion asked; what percentaae of the test items actually 
assess the goal category? If more than 67% of the Items assess behaviors in 
the assigned goal category, the test was credited with two points; if between 
33-67% of the items belonged, the test was credited with one point; and if 
fewer than 33% of the items belonged, the test was credited with zero points. 

The following example illustrates the above procedure. A 50-item test 
was found to have 15 items assessing Agricultural Vocations while the remaining 
35 items were distributed evenly among several other goals. Thus this test was 
assigned to the goal category "Agricultural Vocations". Under Agricultural 
Vocations there are four specific objectives. It was found that three of the 
four objectives were assessed by the 15 items and hence a value of one was as- 
signed for the first criterion. However, the test was assigned a value of 
zerc for the second criterion since the 15 items were less than 33% of the - 
total test items • 

Tests having content validity were defined as those receiving the maximum 
number of points on the two criteria of percentage of goal coverage and percen- 
tage of items belonging in the goal. 

Results 

The survey of the coverage of vocational/career education tests, which are 
appropriate for use with high school students in grades 11 and 12, found a total 
of 1,501 tests and subtests commercially available. The distribution of these 
• 1,501 tests among the various vocational/career education goal categories is 
presented in Table 1. The table also presents content validity data according 
to the two criteria previously described. The number of tests In each goal 
category that assess more (Ng) or less (N^) than 50% of that goal (criterion 1) 
is given. Table 1 further indicates the number of tests (No) that assess more than 
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50% of their goal. arid have more than 67% of their items belonging in the same 
goal (criteria 1 & 2 combined). 



Insert Table 1 about here 

Coverage of Vocational /Career Education Tests 

Because a total of l,5:i tests and subscales were found in the vocational/ 
career education field, it seems safe to say that this field is well-supplied 
liith tests overall. This total would seem to reflect the growing importance 
of vocational and career education in the secondary curriculum. The major 
goal area of General Vocational Interests received the highest percentage of 
assigned tests, 63*02% of the total number of tests. Mechanical-Manual Interests, 
Social Interests, Theoretical Interests and Creative Interests are well covered, 
as each was addressed by more than 118 tests. There were only 10 tests of 
Domestic Interests, the lowest number for any goal in this major area. 

In contrast, the other three areas (i.e.. Knowledge and Skills for Vocational 
Attitudes; General Vocational Aptitudes, General Vocational Attitudes) are 
less well covered* Within these three areas, eight different goals are covered 
by fewer than 10 tests. This does not mean to imply that goals with fewer 
than 10 tests are insufficiently covered, because obviously a single good test 
for any goal would be completely adequate. It simply indicates that in choosing 
a test for a goal in which few tests are available, the counselor encounters 
more restriction of choice than if there had been a greater number of tests to 
choose from at the beginning. 

Content Validity of Vocational/Career Education Tests 

With respect to content validity, a less encouraging situation exists 
when we observe the number of tests that cover more than 50% of their respective 
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goal. Overall, 206 tests, which represent only 13.74% of the total number of 
1,501 tests, satisfy the first criterion of content validity. Additionally, 
in nine goals, none of the tests was judged to cover more than 50% of their 
goal* Only in two goals did a majority of the tests receive full credit on 
the first criterion. These goals were Mathematical Facility and General Voca- 
tional Attitudes with 65.63% and 66.67% of these tests covering more than 
half of their respective goals • The reader should note that the major goal 
area of General Vocational Attitudes is covered by only three tests. 

Regardless of the percentages of tests doing well on the first criterion. 
Table 1 indicates six goals containing 21 or more tests that receive full credit 
on criterion 1. These six goals (i.e.. Clerical Aptitude, Spatial Aptitude, 
Mathematical Facility, Mechanical-Manual Interests, Social Interests, Office 
Interests) are found entirely within two major areas of General Vocational 
Aptitudes and General Vocational Interests and are equally divided between the 
two areas. Even though there exists a considerable number of tests within 
these six goals that satisfy the first criterion of content validity, their 
numbers are so small as compared to the total number published in each goal, 
that they present a problem to the counselor who would hope to select the very 
best ones for possible use. 

The second criterion of content validity was defined as the percentage of 
test items belonging in the appropriate goal category. Only te, ts that satisfied 
criterion 1 were judged on this second criterion. Those tests that did not 
satisfy criterion 1 were not examined further. Tests that received maximum 
credit on both criteria (more than 50% and more than 67% on criteria 1 and 2 
respectively) were defined as possessing good content validity* 

Observing Table 1, we find that the number of tests possessing good content 
validity is 172. This represents 11.45% of the total number of 1,501 tests. 
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In 10 goals not one test satisfies the miniraujii requirements of good content 
validity, while in another 10 goals no more than three content validated tests 
(i.e., meeting criteria 1 and 2) are present. The remaining nine goals are 
represented by nine or more content validated tests all of which are found 
in the two major goal categories of General Vocational Aptitudes and General 
Vocational Interests. The greater number of content validated tests in these 
two particular goal areas is probably due to the fact that more were published 
to begin with and not that they are necessarily constructed better. In those 
10 goals where not one* content validated test is present, there is a clear and 
urgent need for construction of new tests having sufficient content validity. 
These 10 goals are: Knowledge and Skills for Vocational Success, Agricultural 
Vocations, Distributive Vocations, Health Occupations, Technical Vocations, 
General Vocational Aptitudes, Following Directions and Applications, Vocational 
Creativity, Vocational Responsibility, and Career Preparation. 

It is interesting to note that most of the 206 tests satisfying the first 
criterion also received full credit under criterion 2 (83.49%). More speci- 
fically, in 10 out of 19 goals that have tests receiving maximum credit on 
both criteria, al^ tests the passed the first criterion also got the maximum 
score on the second criterion. This fact is illustrated in the last column of 
Table 1 by the 100% entries. This finding suggests that publishers either pay 
full attention to the problem of content validity of ignore it totally. 

Discussion and Conclusions 

According to Super (1953, 1963), as young people think about entering 
an occupation, they seek an occupational role that is reasonably consistent with 
their ideas about the kind of person they are - their interests, abilities, and 
personality traits. Career counseling may be seen as an orderly process of 
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7 • helping people develop and accept an integrated picture of themselves and their 
role" in the world of work, testing this concept against reality, and finally 
In converting it Into a satisfying choice (Herr & Cramor, 1972). 

This view implies that the prime value of vocational/career education tests 
lies, not so much in any ability to predict occupational success, but rather in 
their capacity to provide the necessary information required to help a student 
gain a much more meaningful picture of himself. The adequacy of tne 1,501 
vocational/career education tests and subscales must be gauged by the extent to 
which they actually provide such essential information to high school counselors. 
Tests must have complete congruence with the educational goals they intend to 
assess. Thus, the quality of fit between a test and the particular goal in 
vocational/career education which it purports to measure is of paramount im- 
portance. 

Although the tests provide good overall coverage of the vocational /career 
education goals for secondary education, the poor content validity of the 
majority is disappointing. In most cases this was because a majority of items of 
a given test did not measure the goal of primary interest. The entire test often 
attempted to measure several goals concurrently and as a result did not concen- 
trate on the single goal of primary interest. Counselors should be cautioned 
that ai.^ given vocational /career education tests they intend to employ might not 
be sufficiently valid for their particular purposes. 

In conclusion, the results of this analysis of tests should have immediate 
and long term implications for the improvement of assessment instrumentation by 
pointing out clearly some of the content validity shortcomings that characterize 
today's published vocational/career education tests. Publishers and test con- 
structors should consider more seriously the content validity of their future 
tests. They should analyze each item separately and the test as a whole for 
quality of fit with the specific goal that the test intends to assess. 

ERiC • 
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